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Introduction 


Introduction and Background 


Hand gesture is useful for delivering information that cannot be conveyed 
through speech or type from humans. In this project, we developed a 2D 
hand gesture recognition method which is able to identify simple static hand 
gestures like “stop”, “point”, ’okay” and “up” etc. Hand gesture recognition 
can be applied to the area of security, remote control, and assistance of 
certain groups of disabled people(e.g. translating sign language). Video 
games industry is an early adopter of this technology because it provides a 


more intuitive way for humans to interact with computers. 


Database 


Database 


Our team applies seven categories of hand gestures. They are, namely, 
Okay, Stop, Up, Down, Point, Metal, Vulcan. For each category, we collect 
ten pictures from ten different people and treat these 70 pictures in all as the 
training set. Here we show one of them from each category. 

Okay 


"Okay" Gesture 


Stop 


Up 


"Stop" Gesture 


"Up" Gesture 


Down 


"Down" Gesture 


Point 


"Point" Gesture 


Metal 


"Metal" Gesture 


Vulcan 


"Vulcan" Gesture 


Feature Extraction 


We start with Vulcan! Here is an original picture for our gesture. 
Vulcan 


The Original "Vulcan" Gesture 


Convert image to grayscale. 
Grayscale 


Vulcan converted to grayscale 


Filter out noise and convert the image into a binary image. 
Binary 


Binary Form of Vulcan 


Label the largest connected component in our image matrix and get the 
edge of our gesture. 
Edge 


Edge of Vulcan 


Get polar form of the edge matrix. Divide the polar form into n(here n=8) 
equal bins and count number of 1’s(dots) in each bin. 
Demo 


Bin 5 


/ Center Bin Separation: 45 degrees 


A demonstration of how our algorithm works 


Finally, we compare the data we get here with distribution of dots in our 
training set and get our predicted gesture of this image 


Matlab Code 
Driver.m 


clear all 
close all 
Ti 
I2 
I3 


getnewvector('metal/metal3.JPG', 800, 
getnewvector('metal/metal1.JPG', 800, 
getnewvector('metal/metal2.JPG', 800, 


I4=getnewvector('downi.JPG',800, 700, 1400, 800); 
I5=getnewvector('down2.JPG',800, 700, 1400, 800); 
I6=getnewvector('down3.JPG',800, 700, 1400, 800); 
% need to rescale all images here 


[x, y,1I1]=getcenter(I1); 

[x1, y1,12]=getcenter(I2); 
[x2, y2,13]=getcenter(I3); 
[x3, y3,14]=getcenter(I4); 
[x4,y4,15]=getcenter(I5); 
[x5,y5,16]=getcenter(I6); 


[theta1, ri] = polarize(I1, 
[theta2, r2] = polarize(I2, 
[theta3, r3] = polarize(I3, 
[theta4, r4] = polarize(I4, 
[theta5, r5] = polarize(I5, 
[theta6, r6] = polarize(I6, 


interval=pi/18; 

[ counteri, dividedData1 ] 
interval); 
[ counter2, 
interval); 
[ counter3, 
interval); 
[ counter4, 
interval); 
[ counters, 
interval); 
[ counter6, 
interval); 


dividedData2 |] 
dividedData3 |] 
dividedData4 |] 
dividedData5d | 


dividedData6é |] 


divideIntoBins(theta1, 
divideIntoBins(theta2, 
divideIntoBins(theta3, 
divideIntoBins(theta4, 
divideIntoBins(theta5, 


divideIntoBins(theta6, 


700, 1400, 800); 
700, 1400, 800); 
700, 1400, 800); 


ri, 
r2, 
r3, 
r4, 
r5, 


r6, 


counter2 = counter2/(length(theta2) ) 
counter3 = counter3/(length(theta3) ) 
counter1 = counter1/(length(thetat) ) 
counter4 = counter4/(length(theta4) ) 
counter5 = counter5/(length(theta5) ) 


% figure; 


' 
, 
, 
' 


I 


% subplot(3,1,1); 
% plot(counter1); 
% subplot(3,1,2); 


% plot(counter2); 
% subplot(3,1,3); 
% plot(counter3); 
disp(norm(counteri-counter2, 2) 
disp(norm(counteri-counter3, 2) 
disp(norm(counter1i-counter4, 2) 
disp(norm(counter1-counter5S, 2) 


1 
y 
1 


I 
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%nN1=(norm(counter2,1)+norm(counter3,1))/2 
%N2=(norm(counter4, 1)+norm(counter5, 1)+norm(counter6, 1) )/3 


% 
% % comparing within the same category 


% V1 = getnewvector('vulcani.JPG'); 
% V2 = getnewvector('vulcan2.JPG'); 
% V3 = getnewvector('vulcan3.JPG'); 


% d = zeros(3,1); 

% inner = zeros(3,1); 

% [d(1),inner(1) ] getdistance(vi,v2); 
% [d(2),inner(2) ] getdistance(v2,v3); 
% [d(3),inner(3) ] getdistance(vi,v3); 
% % 


getnewvector.m 


function I=getnewvector(filename, XMIN, YMIN, WIDTH, HEIGHT) 
I=imread( filename) ; 
I=imcrop(I, [XMIN, YMIN, WIDTH, HEIGHT]); %crop 
I=rgb2gray(I); 
filter12=fspecial('gaussian',[20,20],20); %lowpass filter 
I=imfilter(I,filter12); 
% figure; 
% imshow(I); 
%I=imcontour(I,1); 
% magical contour algorithm %get rid of unconnected dots and 
get the largest connect part which is our hand 
I=mean(double(1I)/255,3)>0.5; 
[L,N] = bwlabel(I,4); 
s=zeros(1,N); 
for i=1:N, 
$(i)=sum(L(:)==1); 
end; 
[t,ind]=max(s); 
I=double(L==ind); 


% figure; 
% imshow(I); 


% filter12=fspecial('gaussian',[17,17],17); 
% I=imfilter(I,filteri2); 

figure; 

imshow(I); 

% I = imresize(I, [1601 1601]); 

% I = im2bw(I, .6); 


end 


getcenter.m 


% get the center of the contour image 
% Output the center coordinate, and the edged image 
function [x,y,Iout ]=getcenter(I) 


[m, n]=size(I); 

a=sum(sum(I)); 

rowindex=zeros(a,1); 

colomnindex=zeros(a,1); 

number1=0; 

for i=1:m, 

for j=i:n, 
if I(i,j)==1 
number1=number1+1; 
rowindex(number1, 1)=1; 
colomnindex(number1,1)=j; 
end 
end 

end 

xX=mean(rowindex); 

size(x); 

x=round(x); 

y=mean(colomnindex); 

y=round(y); 

Iout = edge(I, 'canny'); 


end 
processCropped.m 
function [ x1,y1 ] = processCropped( 


category, fileName, saveName ) 
% automate the process of feature extraction 


% Detailed explanation goes here 
dir = strcat(category,'/',fileName); 
I = getnewvector(dir,0, 0, 1601, 1601); 
[x1,y1,I] = getcenter(I); 


dir2 = strcat(category,'/',saveName); % save directory 
imwrite(I,dir2, 'bmp'); 


end 


processCategory.m 


function processCategory( category, numOfPics, saveAbrev ) 
% process the entire category 
% Detailed explanation goes here 
center = zeros(numOfPics, 2); 
for n = 1:numOfPics, 
fileName = strcat(category,num2str(n),'.jpg'); 
saveName = strcat(saveAbrev,num2str(n),'.bmp'); 
center(n,:) = processCropped( category, fileName, saveName) ; 
end 
csvwrite(strcat(category,'/', 'centers.dat'),center); 
end 


polarize.m 


function [theta, r] = polarize(binaryMatrix, center) 
%POLARIZE Summary of this function goes here 
% Center: 1x2 
[m,n] = size(binaryMatrix); 
theta = zeros(500,1); 
r = zeros(500,1); 
numOfDots = 0; 
for i= 1:m, % for each row 
for j = 1:n, % for each column 
if (binaryMatrix(i,j) == 1), 
numOfDots = numOfDots + 1; 
dot = [i,j] - center; % find the relative coordinate 
with centroid 
[theta(numOfDots), r(numOfDots)] = 
cart2pol(dot(1),dot(2)); % convert into polar 
end 
end 
end 
end 


divideIntoBins.m 


function [ counter, dividedData ] = divideIntoBins(theta, r, interval) 
% Divide into Bins 
% interval: radian specified to divide the polar coordinated image 
% needs to be divisible by 2pi 
% theta: angles for each dot with respect to centroid, column vector 
% r: distance of each dot with respect to centroid, column vector 
bins = -pi:interval:pi; 
% now we need a counter for each bin, 1xlength(bins) 
% and a container for the different r in the bin 
counter = zeros(1,2*pi/interval); 
dividedData = zeros(40,2*pi/interval); % each column represent a bin 


len = sizeCtheta); 
for n = 1:len(€1), % for every dot's angle 
for k = 1:length(bins)-1 % look through the bins and process it. 
if(bins(Ck) <= theta(n) && theta(n) < bins(k+1)) 
counter(k) = counter(k) + 1; % increment the counter in that bin 
dividedDataCcounter(k),k) = r(n); % store the Length of the dot in the column 
break; 
end 
end 
end 


end 


testDriver.m 


clear all 

close all 

dir =char('down', 'up', 'okay','metal', 'point', 'stop', 'vulcan'); 
right=0; 

wrong=0; 

for n=1:7, 


traningi=csvread(strcat(dir(n,:),'/', 'features.dat')); 
[x y]=size(traning1); 
for k=1:x, 

training=traning1(k,:); 


type=classify(training,2,7,8); 


type1=mode(type); 


if type1==n 


right=right+1; 
else 
wrong=wrong+1; 
end 
end 


end 
size(type) 


right 
wrong 


% 
% 
% 
% 
% 
% 
% 
% 
% 
% 
% 
% 
% 
% 
% 
% 
% 
% 
% 
% 
% 
% 


I1 = getnewvector('metal/metal3.JPG',800, 700, 1400, 
I2 = getnewvector('metal/metali.JPG',800, 700, 1400, 
I3 = getnewvector('metal/metal2.JPG',800, 700, 1400, 
I4=getnewvector('downi.JPG',800, 700, 1400, 800); 
I5=getnewvector('down2.JPG',800, 700, 1400, 800); 
I6=getnewvector('down3.JPG',800, 700, 1400, 800); 

% need to rescale all images here 


[x, y,11]=getcenter(I1); 

[x1, y1,12]=getcenter(I2); 

[x2, y2,13]=getcenter(I3); 

[x3, y3,14]=getcenter(I4); 
[x4,y4,15]=getcenter(I5); 
[x5,y5,16]=getcenter(I6); 

[theta1, ri] polarize(I1i, [x,y]); 
[theta2, r2] polarize(I2, [x1,y1]); 
[theta3, r3] polarize(1I3, [x2,y2]); 
[theta4, r4] polarize(1I4, [x3,y3]); 
[theta5, r5] polarize(1I5, [x4,y4]); 
[theta6, r6é] polarize(I6, [x5,y5]); 
interval=pi/18; 

[ counter1, dividedData1 |] = divideIntoBins(theta1, 


interval); 


% 


[ counter2, dividedData2 |] = divideIntoBins(theta2, 


interval); 


% 


[ counter3, dividedData3 |] = divideIntoBins(theta3, 


interval); 


% 


[ counter4, dividedData4 ] = divideIntoBins(theta4, 


interval); 


% 


[ counter5, dividedData5 |] = divideIntoBins(theta5, 


interval); 


% 


[ counter6, dividedData6 |] = divideIntoBins(theta6, 


interval); 


% 
% 
% 
% 


counter2 = counter2/(length(theta2) ); 
counter3 = counter3/(length(theta3) ); 
counter1 = counter1/(length(thetat1)); 
counter4 = counter4/(length(theta4) ); 


800); 
800); 
800); 


ri, 
r2, 
r3, 
r4, 
r5, 


r6, 


% counter5 = counter5/(length(thetad) ); 
% % figure; 

% % Ssubplot(3,1,1); 

% % plot(countert1); 

% % Ssubplot(3,1,2); 

% % plot(counter2); 

% % Ssubplot(3,1,3); 

% % plot(counter3); 

% disp(norm(counter1-counter2, 2) 
% disp(norm(counter1-counter3:, 2) 
% disp(norm(counter1-counter4, 2) 
% disp(norm(counter1-counter5, 2) 
% 

% %N1=(nNorm(counter2,1)+norm(counter3,1))/2 

% %N2=(norm(counter4, 1)+norm(counter5,1)+norm(counter6, 1) )/3 
% 

% 

% 

% % 

% % % comparing within the same category 


1 
1 
y 


y 
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% % V1 = getnewvector('vulcani.JPG'); 
% % v2 = getnewvector('vulcan2.JPG'); 
% % V3 = getnewvector('vulcan3.JPG'); 


%% d = zeros(3,1); 

% % inner = zeros(3,1); 

% % [d(1),inner(1) ] getdistance(vi,v2); 
% % [d(2),inner(2) ] getdistance(v2,v3); 
% % [d(3),inner(3) ] getdistance(vi,v3); 
% % % 


classify.m(part1) 


function type=classifyCinvector, normsize ,numOfFeatures ,minnum) 
type=zeros(minnum,1); 
outmatrix=0; 
dir =charC'down','up', ‘okay’, ‘metal’, ‘point’, ‘stop’, ‘vulcan'); 
Labelvect=zeros(numOfFeatures,1);% see which index is in which category 
for n=1:num0fFeatures, 
filename= strcat(dir(n,:),'/','features.dat'); 
trainingmatrix=csvread( filename); 
(x, yJ=sizeCtrainingmatrix); 
labelvect(n,1)=x; 
for i=1:x, 
outmatrixCend+1, :}=norm(trainingmatrix(i, :)-invector,normsize); 
end 
end 
{m g]=sizeCoutmatrix) 
outmatrix 


indexvec=zeros(minnum,1); 
outmatrixl=sort(outmatrix,'ascend') % find the least norms 
for j=1:minnum, 
for k=1:m, 
if outmatrix1(Cj, :)==outmatrixCk, :) 
indexvec(j, :)=k; 
end 
end 
end 
labelvect=cumsum(Labelvect); 
indexvec 
labelvect 


classify.m(part2) 
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Introduction & Background 


information that cannot be conveyed 
through speech or type from humans. 
In this project, we developed a 2D 
hand gesture recognition method 
which is able to identify simple static 
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hand gestures like ‘stop’, “point’, 
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okay” and “up” ete, 

Hand gesture recognition can be 
applied to the area of security, 
remote control, and assistance of 
certain groups of disabled people(e.g, 
translating sign language). Video 
games industry is an early adopter of 
this technology because it provides a 
ore intuitive way for humans to 
interact with computers, 
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Hand gesture is useful for delivering 
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Figure 1 


Summary of Methods 


We use edge detector to obtain the 
edge of the gesture and collect 
features of the image to construct our 
training data, Then we compare the 
test image to our training data and 
select the closest gesture among our 
7 gestures. 


Features f9"Poit’ Gestures 


Features of 10 "Vulcan Gestures 


Results 


Bins Seowated 10 Degrees: 


gestures. 


5) Repeat this procedure for all 7 


3) We find the 


Figure 2 


Bin 4 


/ Bin’? 


Processing Images Making Prediction 
| 
Edge Extraction Feature Extraction & Prediction 
1) Take a group of training images for 1) As shown in Figure 5, we first divide the 
each gesture, one of whichis the edge matrix into n equal bins and count a; , 
“Vulcan” gesture. (Figure 1 the number of 1s in each bin, by obtaining 
1} Convert to grayscale, (igre 2 apolar form of the edge matrix. Let feature 
tor f = {yd 

3) Filter out noise and convert the sic : ( fe tr) ; 
: . . 2) For an input image, we extract its feature 
image to a binary matrix. Label and crop Sey a 
out the largest connected component in vector finputlizng the same method and 
the binary matrix. (Figure 3 calculate d , the distances from finput to all 
4) Obtain the edge matrix. (Fig. 4) feature vectors of training images. 


k minimum values in d, and 


compute the mode of them, which gives us 
the most likely gesture of the image, 


Bin Separation: 45 degrees 


Bin 4 


Bin 


Correct Rate for Training Set 
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Comect Rate for Test Set 


Incorrect 


Number of Corect Recognitions for Each Gesture 


0 
om up hay real pint stop wlean 


Program Flow 


Sample Gestures | 
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Detector 


Feature 
Extraction 


Feature 
Analysis 


Compare to 
Training data 


Derive 
Conclusion 


Future Work 


1) Explore other 
features such as 
shape context, 
HOG or classifiers 
such as SVM. 

2) Expand the work 
content into motion 
gesture detection 
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Results 


Number of Correct Recognitions for Each Gesture 
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Number of Correct Recognitions for Each Gesture 


Correct Rate for Test Set 
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